Selection-Based Language Model for Domain Adaptation using Topic Modeling

This paper introduces a selection-based LM using topic modeling for the purpose of domain adaptation which is often required in Statistical Machine Translation. The performance of this selection-based LM slightly outperforms the state-of-theart Moore-Lewis LM by 1.0% for EN-ES and 0.7% for ES-EN in terms of BLEU. The performance gain in terms of perplexity was 8% over the Moore-Lewis LM and 17% over the plain LM. 1 Domain Adaptation in Statistical Machine Translation Domain adaptation is one important research area in Statistical Machine Translation (SMT) as well as other areas of NLP such as parsing. Domain adaptation tries to ensure that the performance is not radically decreased even if we translate a text in a test set whose genre is different from the parallel corpus which is used to build the system. Without loss of generality, the decoder of an SMT system can be written in the form of the noisy channel model argminE P (E|F )PLM (E) where the two components, the set of P (E|F ) and that of PLM (E), are the targets that we do domain adaptation on: the set of P (E|F ) is called a phrase table (or a rule table) and that of PLM (E) is called a language model (for simplicity, the model is written in the simplest form without indices). Hence, one approach to domain adaptation in SMT aims at obtaining a domain-adapted phrase table and language model [1]. In particular, it is observed in several papers that the domain adaptation of the language model is often the most effective route in domain adaptation. In this context, we explore domain-adapted language models using topic modeling [2] in this paper. Note that there is an alternative approach which applies transfer learning [3] for domain adaptation, which is not pursued in this paper. In the following, we focus on the domain adaptation of language models and we leave the topic of translation model domain adaptation as further work. The special setting for SMT would be the following: (1) there is a tendency that if a training corpus becomes big, e.g. more than a million sentences, we may need to think about the corpus as a combination of different genres, and (2) we may have some information about the genre of a test set as a whole or for each sentence (it is rare that we do not have any information about the genre of the test set). 2 Selection-Based Language Models Let us prepare n kinds of language models {PLM1 , . . . , PLMn} (we sometimes call this “a pool of language models” in the following) and a selection function f(s)where s denotes a test sentence and The topic modeling for translation model can be found in [4]. Main differences are the usage of crossentropy and interpolation. The topic modeling for system combination in SMT can be found in [5].